The Benefit of Additional Opinions

In daily decision making, people often solicit one another’s opinions in the hope of improving their own judgment. According to both theory and empirical results, integrating even a few opinions is beneficial, with the accuracy gains diminishing as the bias of the judges or the correlation between their opinions increases. Decision makers using intuitive policies for integrating others’ opinions rely on a variety of accuracy cues in weighting the opinions they receive. They tend to discount dissenters and to give greater weight to their own opinion than to other people’s opinions. KEYWORDS—judgment and decision making; aggregating opinions; combining information It is common practice to solicit other people’s opinions prior to making a decision. An editor solicits two or three qualified reviewers for their opinions on a manuscript; a patient seeks a second opinion regarding a medical condition; a manager considers several judgmental forecasts of the market before embarking on a new venture. All these situations involve the decision maker in the task of combining other people’s opinions, mostly so as to improve the final decision. People also seek advice when they feel strongly accountable for their decisions. An accountant performing a complex audit might solicit advice to help justify his or her decisions and share the responsibility for the outcome with others. One could justifiably argue, however, that even such reasons for seeking others’ opinions are rooted in the belief that this process could improve decision making. Two main questions arise in the research on combining opinions. One involves the statistical aspects of the combination task: Under what conditions does combining opinions improve decision quality? The other concerns the psychological process of combining judgments: How do judges utilize other people’s opinions? These questions, which have been investigated by students of judgment and decision making, statistics, economics, and management, are intertwined, because the quality of the product is related to the way it is produced. In this review, I discuss what researchers have learned about the process and outcomes of combining opinions. Our focus here is on situations in which a decision maker seeks quantitative estimates, judgments, and forecasts from people possessing the relevant knowledge. The opinions are then combined by the individual decision maker, not by a group (decision making in groups deserves a separate discussion). It is useful to distinguish between two ways in which expert judgments can be combined: (a) intuitively (subjectively) and (b) mechanically (formally), that is, by using a consistent formula, such as simple or weighted averaging. ACCURACY GAINS FROM AGGREGATION Research has demonstrated repeatedly that both mechanical and intuitive methods of combining opinions improve accuracy. For example, in a study of inflation forecasts, the aggregate judgment created by averaging the forecasts of expert economists was more accurate than most of these individual forecasts, though not as good as the best ones (Zarnowitz, 1984). The best forecasts, however, could not be identified before the true value became known. Hence, taking the average was superior to selecting the judgment of any of the individuals. Moreover, a small number of opinions (e.g., three to six) is typically sufficient to realize most of the accuracy gains obtainable by aggregation. These fundamental results have been demonstrated in diverse domains, ranging from perception (line lengths) and generalknowledge tasks (historical dates) to business and economics (sales or inflation forecasts), and are an important reason for the broad interest in research on combining estimates (Johnson, Budescu, & Wallsten, 2001; Sorkin, Hayes, & West, 2001; Yaniv & Kleinberger, 2000). How Does Combining Opinions Improve Judgment? The improvement in accuracy is grounded in statistical principles, as well as psychological facts. For quantitative estimates, a common measure of accuracy is the average distance of the prediction from the event predicted. In the special case of judgments made on an arbitrary rating scale (e.g., an interviewer’s rating of a job candidate’s capability on a 9-point scale), a common measure is the correlation between the judgments and some objective outcome (e.g., the candidate’s actual success). In the case of quantitative estimates, it can be outlined in simple terms why improvement is to be expected from combining estimates. A subjective estimate about an objective event can be viewed as the sum of three components: the ‘‘truth,’’ random error (random Address correspondence to Ilan Yaniv, Department of Psychology, Hebrew University, Jerusalem 91905, Israel; e-mail: ilan.yaniv@ huji.ac.il. More complex methods based on Bayes’s theorem are less common in psychological research on combining opinions; hence, they are not treated here. CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE Volume 13—Number 2 75 Copyright r 2004 American Psychological Society fluctuations in a judge’s performance), and constant bias (a consistent tendency to overor underestimate the event). Statistical principles guarantee that judgments formed by averaging several sources have lower random error than the individual sources on which the averages are based. Therefore, if the bias is small or zero, the average judgment is expected to converge about the truth (Einhorn, Hogarth, & Klempner, 1977). The case of categorical, binary judgments (e.g., a physician inspects a picture of a tumor and estimates whether it is benign or malignant) requires a special mention. Suppose a decision maker polls the judgments of N independent expert judges whose individual accuracy levels (chances of choosing the correct answer) are greater than 50% and then decides according to the majority. For example, three experts might judge whether or not a witness is lying, and the final decision would be the opinion supported by two or more experts. According to a well known 18th-century theorem (known as Condorcet’s jury theorem), the accuracy of the majority increases rapidly toward 100% as N increases (e.g., Sorkin et al., 2001). Thus, the majority outperforms the individual judges. For instance, the majority choice of five independent experts who are each correct 65% of the time is expected to be correct approximately 76% of the time. Conditions Under Which Accuracy Gains Are Observed A central condition for obtaining optimal accuracy gains through aggregation is that the experts are independent (e.g., little gain is expected if judge B is essentially a replica of judge A). But gains of appreciable size can be observed even when there are low or moderate positive correlations between the judgments of the experts (Johnson et al., 2001). The gains from aggregating quantitative judgments are also determined by the bias and the random error of the estimates (the lower the better). If judgments are made on rating scales, then the accuracy gains are related directly to the validity of each judge (i.e., how the judge’s ratings correlate with the objective value of what is rated) and indirectly to the correlations between different judges’ ratings (Einhorn et al., 1977; Hogarth, 1978; Johnson et al., 2001). Number of Opinions Needed As already noted, as few as three to six judgments might suffice to achieve most of what can be gained from averaging a larger number of opinions. This puzzling result that adding opinions does not contribute much to accuracy is related to my previous comments. Some level of dependence among experts is present in almost any realistic situation (their opinions tend to have some degree of correlation for a variety of reasons—they may rely on similar information sources or have similar backgrounds, or simply consult one another; cf. Soll, 1999). Therefore, the benefits accrued from polling more experts diminish rapidly, with each additional one amounting to ‘‘more of the same.’’ Similarly, bias or low judge validity limits the potential accuracy gains and further diminishes the value of added opinions. PSYCHOLOGICAL EFFECTS ON THE AGGREGATION OF